@njsmith: this change would have the CI hitting your vorpus.org server - the code was there before, but commented out. I'd like your approval before merging it.

@rmcgibbo: your 'wheelhouse' at stanford.edu is no longer available. Let me know if you have the same files at some other URL. I don't think this is high priority, so don't spend energy on it, but if it's just a case of tweaking the URL, we can do that.

I don't have those files anymore (Stanford took down my webspace a couple months after I graduated), but they weren't anything special.

@takluyver: from these these travis logs from 3 years ago, it looks to me like the tests were covering wheels from lxml-3.7.3, Pillow-4.1.1, PyYAML-3.12.

Thanks! Do you think it's worth re-creating some more tests with real-world wheels, or is it more a nice-to-have alongside the main test suite? I wanted to make sure that whatever's there is running effectively, but I don't know how important it is.

This should fix #142. We should also remove the allow_failures bit in .travis.yml.

The latest commit removes the allow_failures section.

I'm in two minds about this. On the one hand, it could be a flaky test, as it relies on downloading extra files. But then I don't think allow_failures tests are much use. At least the way I use Travis, I usually ignore them, and as these have been failing for about 18 months, it seems I'm not the only one.

In any case, I'd still like an OK from @njsmith before we merge this, because it will hit his server.

Do you think it's worth re-creating some more tests with real-world wheels, or is it more a nice-to-have alongside the main test suite? I wanted to make sure that whatever's there is running effectively, but I don't know how important it is.

Well, I think that's the main question here. I don't know if it's important either... Were those tests added to cover a feature, bug, in the past, I can't say looking at the history...

I added those tests, but it was 3 years ago so I don't remember my exact train of thought. Generally though, at that time this project was something I was just hacking on. I don't think it was an official pypa project, I don't think that the first manylinux pep had yet been merged (it was probably under discussion), and it wasn't clear whether this software would even fill the intended use case. There wasn't really a "main test suite". I think I basically figured "if I can process some random wheels that I found on the internet, then it's probably working, and that's easier than investing in writing a bunch of unit tests right now"

So... real-world tests are always great. But I'd recommend making the decision on whether or not to include tests like this based on your current assessment of the coverage of the test suite, not some no-longer-applicable-determination I made three years ago when this was a solo hobby project that I was hacking on while procrastinating on my dissertation.

Thanks @rmcgibbo. That's the sort of scenario I would have guessed that script came from. :-)

@lkollar is doing some good work towards increasing test coverage. But I don't yet have a good feel for how well the test suite exercises gnarly cases that might come up in real-world wheels.

Given what @rmcgibbo said it might not be useful to try to fix these tests with the same wheels. The test-wheelhouse.sh script does not try to import the repaired wheels either, so this test is not that useful anyway. The repair step's return value is not even checked so we won't catch issues there. We now have several more exhaustive tests which validate specific features and actually test that the resulting libraries are usable. So I think we should consider two alternatives here:

• Remove these tests altogether and keep adding integration tests for wheels for the gnarly cases. Python C++ module not properly repaired, retains dependency on symlink to bundled library #159 is an example but we would have to xfail it until patchelf lands the fix. I'm sure we could find more in the issue tracker.
• Keep the wheels but replace the shell script with a proper Python test which repairs and imports the modules.

If we keep the wheel I still think that we should store them in the source repo. pip does this too and it seems to be working just fine.

Now we've got coverage measurement set up and we can see we're exercising most of the code already, I think I'm comfortable with removing the wheelhouses script, and #179 does that.

#179 has been merged instead of this.